1. Field of the Invention
The present invention relates to vehicle environment models for driver assistance systems. In particular, the present invention relates to generating camera-based vehicle environment models for real time applications.
2. Related Art
Contemporary vehicles are equipped with a series of sensors. Vehicle sensors include sensors of a first kind for detecting variables that are related to the status of the vehicle itself, as well as sensors of a second kind for detecting variables of the environment of the vehicle.
Examples of first type sensors include sensors for motor rotation speed, brake status, throttle position, vehicle speed and acceleration, steering position, tire pressure, occupant position, seatbelt position, passenger airbag status and others. Sensors of the second type include temperature sensors, distance sensors, antennae of telematic and navigation systems and others.
In particular, in modern practice, it is becoming more and more common to also equip vehicles with cameras. A vehicle can be equipped with a single or a plurality of cameras mounted at different positions inside and/or outside the vehicle. Cameras mounted inside a vehicle can be employed for monitoring objects and events occurring inside the vehicle, and are specifically used in vehicles of public transportation, such as buses and coaches. Cameras for monitoring the environment of the vehicle are specifically designed to capture images of a certain sector of a vehicle's environment.
Data obtained from sensors of a vehicle, including cameras, are employed for a variety of purposes. A basic class of functions, for which it is necessary to collect and further process sensor data, is the field of driver assistance systems. Driver assistance systems known in the art cover a large range of functions. Systems exist that simply provide a driver with particular information, including warning in the case of possible emergency situations inside or outside the vehicle. More sophisticated driver assistance systems enhance a driver's comfort by interfering with or partly taking over control functions in complicated or critical driving situations. Examples for the latter class of driver assistance systems are antilock brake systems (ABS), traction control systems (PCS), and electronic stability programs (ESP). Further systems that are currently under development and do not yet belong to default vehicle equipment include adaptive cruise control, intelligent speed adaptation and predictive safety systems.
Furthermore, vehicle sensors are employed for environment monitoring for purposes other than vehicle safety and control, such as for patrol vehicles monitoring parked cars or for recording in event data recorders. Event data recorders (EDR, also known as “black box”) record internal and external information gathered by various sensors of a vehicle to enable a reconstruction of events that have occurred in the last few seconds or minutes immediately before an accident.
FIG. 1 illustrates an example of a vehicle (a car 100) that is equipped with a plurality of sensors, including a camera 102a and further sensors 102b and 102c. The car is moreover equipped with an antenna 104, which enables reception and transmission of data, including, for instance, those of a satellite navigation system. Sensor data is forwarded to a processing unit 106, where the data are processed to generate a response. A response generated by the processing unit 106 may include signals for triggering any driver assistance functions, including those commonly employed by the above mentioned driver assistance systems. In the simplest case, the processing unit issues signals comprising information to be notified to a driver. In particular, notification can be issued for display on a display device 108 that is arranged near the driver's seat.
FIG. 2 is a principal block scheme of a conventional driver assistance system. Data collected by a plurality of sensors (s1, s2, s3, . . . , sn in FIG. 2) are fed into processing unit 106. Processing unit 106 generates a response on the basis of processing the data received from the sensors. The response includes signals that are either forwarded to a device for notifying the driver, or to specific units for particular control functions.
In the particular case of employing cameras, a rather complete information of a section of a vehicle's environment can be obtained by capturing an image. By employing several cameras, a range of a vehicle's environment covers all directions, rather than only a particular section, such as a section in forward direction. Cameras mounted to a vehicle include two-dimensional (2D) and three-dimensional (3D) cameras. While 3D-cameras are capable of capturing a three-dimensional image in one shot, by employing two separate optical systems mounted adjacent to each other, in a similar manner as a stereoscopic view is achieved with the help of a pair of human eyes, three-dimensional environment information of a moving vehicle can also be obtained by only employing a single two-dimensional camera. Therefore, additional information from sensors detecting a moving speed and direction, as well as changes in vehicle orientation are employed. On the basis of the additional data, an approximate three-dimensional environment model can be generated, by evaluating changes between two-dimensional images that have been captured at subsequent instances of time with respect to said detected vehicle motion parameters.
Although cameras are generally capable of providing rather complete information, the employment of cameras as sensors suffers from having to provide a large amount of information that is redundant or irrelevant in a particular situation, or in view of the particular purpose for which the information is to be used. Accordingly, a large processing time is required to process the large volume of data included in images captured by cameras. Therefore, in the case of employing cameras as sensors of a vehicle, the advantage of obtaining more complete information compared to a case where only specific sensors are employed, goes hand in hand with a drawback of large processing times required for obtaining a response to the received data. However, specifically in the case of driver assisting systems utilized to issue a warning to avoid a threatening emergency situation, or to trigger active and/or passive countermeasures, the importance of processing time is crucial.
A possibility of improving the described situation, and achieving lower processing times, may be achieved by performing a pre-processing procedure of captured image data. In this regard, to reduce the amount of data to be processed for responding, only particular data are extracted from an image. Such pre-processing steps may include, for instance, filtering, rasterization and vectorisation.
It is, however, a drawback of employing pre-processed information from images captured by cameras attached to a vehicle because the pre-processing affects the information included in the images in a predefined, static manner. The pre-processing steps are adapted to specific requirements in accordance with particular situations in which a camera is utilized for environment monitoring. Such specific situations and the respective circumstances under which the captured information is to be utilized, includes, however, a large range having their own very specific and different requirements. For example, environment monitoring by cameras can be used to avoid potential collisions when driving on a motorway at a high speed, and in dense traffic. On the other hand, cameras of a vehicle may assist the driver in parking the vehicle, in a small amount of space. While in the first case only rather general information concerning other vehicles in the immediate vicinity of the vehicle are necessary, the time available for taking a decision is extremely small, i.e., in the range of a few seconds. In the second case, processing time is not as crucial as in the first case, but the assistance will be more helpful, the more complete the available information.
Another case that is particularly influenced by appropriate pre-processing concerns storage in an event data recorder. The storage capacity of an event data recorder is generally limited. Therefore, the depth of the history that can be stored in an event data recorder (i.e., the time period immediately before an accident, for which the event data recorder can store data) considerably depends on the amount of data to be stored representing the overall (external and internal data) vehicle situation at each single instance of time. On the other hand, the event data recorder does not require the inclusion of complete image information, but rather only information concerning position, as well as absolute value and direction of the velocity of the vehicle itself, and all objects and subjects in the immediate neighborhood of the driven vehicle. Accordingly, a need exists to provide an improved system for more efficiently obtaining environment monitoring information of a camera-equipped vehicle.